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Technology Alignment and Portfolio Prioritization (TAPP) is a method being 
developed by the Advanced Concepts Office, at NASA Marshall Space Flight Center. 
The TAPP method expands on current technology assessment methods by 
incorporating the technological structure underlying technology development, e.g., 
organizational structures and resources, institutional policy and strategy, and the 
factors that motivate technological change. This paper discusses the methods ACO is 
currently developing to better perform technology assessments while taking into 
consideration Strategic Alignment, Technology Forecasting, and Long Term 
Planning. 


I. Introduction 

The Advanced Concepts Office (ACO) at NASA, Marshall Space Flight Center is expanding its current 
technology assessment methodologies. ACO is developing a new framework called TAPP that uses a variety 
of methods, such as association rule learning from data mining, analysis of technology system structures 
using a Technological Innovation System (TIS), and social network modeling to measure structural 
relationships between multiple technological systems. The reason ACO performs technology assessments are 
to 1) produce a broad spectrum of ideas and alternatives for a variety of NASA’s missions, 2) determine 
mission architecture feasibility and appropriateness to NASA’s strategic plans, and 3) define a project in 
enough detail to establish an initial baseline capable of meeting mission objectives 

ACO’s role supports the decision-making process associated with the maturation of concepts for traveling 
through, living in, and understanding space. ACO performs concept studies and technology assessments to 
determine the degree of alignment between mission objectives and new technologies. 

The first step in technology assessment is to identify the current technology maturity in terms of a 
technology readiness level (TRL). The second step is to determine the difficulty associated with advancing a 
technology from one state to the next state. 1 

NASA has used TRLs since 1970 2 and ACO formalized them in 1995. 3 Many government and 
commercial industries use modified TRL definitions to perform technological assessments. The DoD, ESA, 
Oil & Gas, and DoE have adopted TRLs as a means to assess technology maturity. However, “with the 
emergence of more complex systems and system of systems, it has been increasingly recognized that TRL 
assessments have limitations, especially when considering [the] integration of complex systems. 2 ” 

When performing the second step in a technology assessment, NASA requires that an Advancement 
Degree of Difficulty (AD2) method be utilized. NASA has developed and used a variety of methods to 
perform this step: Expert Opinion or Delphi Approach, Value Engineering or Value Stream, Analytical 
Hierarchy Process (AHP), and other multi-criteria decision making methods. 4 These methods can be labor- 
intensive, contain parochial bias, and seldom consider the competing prioritization between mission 
architectures. 
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Assessing technological change is particularly challenging due to a multitude of relationships. The central 
idea in technology dynamics is to consider all activities that contribute to the development, diffusion, and 
use of innovations as system functions. 5 Bergek, in Ref. 5, defines these system functions within a TIS to 
address what is actually happening and identify the technologies that have a direct influence on the ultimate 
performance of the system and technology development. ACO uses similar metrics and is expanding these 
metrics to account for the structure and context of the technology. 

At NASA technology, strategy and policy is strongly interrelated. NASA’s Strategic Space Technology 
Investment Plan (SSTIP) prioritizes technologies essential to the pursuit of NASA’s missions and national 
interests. The SSTIP is coupled to NASA’s Technology Roadmaps, which provide investment guidance 
during the next four years, within a twenty-year horizon. 6 

This paper discusses the methods ACO is currently developing to better perform technology assessments 
while taking into consideration Strategic Alignment, Technology Forecasting, and Long Term Planning. 

II. Background 

“ Measuring technological change is difficult. 7 ” 

There are two reasons for this statement. Giovanni Dosi in Ref. 8, Technology Paradigms and 
Technology Trajectories, states “technological knowledge is much less articulated than scientific knowledge” 
and technological paradigms* “have a powerful exclusion effect since they are often focused on precise, 
prescribed directions with an associated “momentum of its own”, or a natural trajectory of technological 
progress." 

Technology Paradigm is not the same as Scientific Paradigm 
Technological Trajectories t have a natural bias built into the direction selected. 

In Ref. 9, Technological Revolutions and Techno -economic Paradigms , Carlota Perez reiterates these 
points as well, saying that a technological paradigm represents “the tacit agreement of the agents involved as 
to what is a valid search direction and what will be considered an improvement or a superior version of a 
product, service or technology.” Technological Trajectories are influenced by policy, institutional structures 
and policies, and interrelated technological systems. Technological progress is incremental. Perez 
hypothesizes that a technological revolution ^ occurs when there is a strong interconnectedness between 
technological systems with the capacity to transform profoundly the rest of an economy. 

III. Methods of Measuring the State of Technology and Technological Trajectories 

There are many papers written on the processes, methods and tools necessary to perform a Technology 
Assessment and the trajectories available to the technology. Table 1 provides a few representative methods 
used by ACO. Common to the methods listed are dependency structures in the form of matrices and networks 
that model the relationship between objects of interest. While each method has its strength, each method also 
has a shortcoming. This does not imply any particular method is bad, it implies each method has a focused 
purpose. Ideally a Technology Assessment should be able to span across a variety of methods, adjusting the 
fidelity, scale and resolution of the analysis to accommodate the task at hand. 


Table 1 Representative Methods, Processes and Tools for Performing Technology Assessment 


* Technological Paradigm: a "model" and a "pattern" of solution of selected technological problem based 
on selected principles derived from natural sciences and on selected material technologies. 8 

^ Technological Trajectory: the pattern of "normal" problem solving activity (i.e. of "progress") on the 
ground of a technological paradigm. 8 

* Technological revolution: a set of interrelated radical breakthroughs, forming a major constellation of 
interdependent technologies; a cluster of clusters or a system of systems. 9 
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Year 

Objective 

Shortcomings 

Primary References 

1970- 

1990 

Measuring the 
state of 
technology 
through 

readiness levels 

Assumes technologies exist 
independently, does not consider the 
technological system or system 
architecture 

Qualitative, subjective and parochial 
bias 

Ref. 3: Technique based on 
defining the state of a 
technology 

1995- 

2005 

Optimization 

Multi-Criteria 

Decision 

Analysis 

(MCDA) 

Technology Trajectory is design 
focused, surrogate model building time 
consuming process 
Assumes technologies exist 
independently with linear evaluation 
criteria 

Ref. 10: Technique based on 
multi-attribute decision-making 
and morphological matrices 

2002- 

2010 

Technology 
Gap and 
Prioritization 

Labor Intensive, requires support from 
expert technologist 
Technology trajectory is design 
focused 

Ref. 4: Technique based on 
interviews and Delphi 

2005- 

2010 

Technological 

Change 

Ability to adequately model the 
institutional and technological 
structures 

Ability to obtain information is labor 
intensive 

Ref. 5: Assessment based on 
assessment of a variety of 
readiness levels, not uniformly 
applied 

Ref. 12: Adds Puzzy Analytical 
Hierarchy Approach (PAHP) 

2011- 

Technology 

Innovation 

System 

Analysis 

Policy and Institutional focused 
Performance metrics 

Ref. 13: Network of interacting 
agents 

Ref. 14: Complex system and 
functional dependencies 
networks and relationships 


In Ref. 15, Technology Readiness Levels at 40: A study of the state of the Art: Use, Challenges and 
Opportunities, Olechowski, Eppinger, and Joglekar evaluate various methods of performing Technology 
Assessment and describes 15 challenges relating to the use of TRL in practice today. The results of these 
findings are grouped into three categories: 

• System Complexity 

7 challenges related to the complexity of the system, whether it is at the component level, 

architectural level or system as a whole (across missions) 

• Planning and Review 

6 challenges related to the decision process and technological progression 

• Assessment Validation 

2 challenges related to quality of the information, subjectivity and imprecision 

IV. Previous Technology Assessments and Methodologies 

In an era of reduced funding and increasing mission/vehicle complexity, NASA needs tools and 
methodologies to rapidly and accurately assess the benefits and costs of technologies that may enhance or 
enable its mission. As early as 2002 NASA had begun developing tools and methodologies to rapidly and 
comprehensively assess technologies for integrated mission architectures. The first of these and one of the 
most comprehensive was for the Second Generation Reusable Launch Vehicle Space Launch Initiative (2nd 
Gen RLV SLI). NASA formed a multi-center team called the Intercenter Systems Analysis Team (IS AT) 
lead by the Program Development Office at the Marshall Space Plight Center. This team conducted the 
technology and architecture assessments. The team’s objectives were 1) determine the impact of individual 
technologies on reference reusable launch vehicle architectures, 2) identify technologies which provide the 
most benefit to launch architectures, 3) evaluate combination of technologies in conjunction with one another, 
and 4) analyze the affects of sequentially adding funded technologies to a reference concept. The process 
developed to perform these assessments is shown in Error! Reference source not found.. 
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Key to this process is an understanding of 
the program/Agency goals (Program 
Requirements), the collection of the 
technology data, and the actual technology 
assessments. The process requires 
significant input in the form of interviews 
with technologist throughout the 
industry /government /academia. Also, the 
technology assessments require systems 
assessments using multiple discipline 
analysts and multiple systems design tools. 
These processes are both laborious and 
time intensive due to the nature of the 
information technology of the time and the 
available information. Each of which 
could lend itself to automation. The 
requirements of this process, while 
extremely beneficial, could be limiting due 
to time and cost constraints. The results of 
the study were technology suites that were 
selected based on a number of figures-of- 
merit such as performance, cost, and 
reliability. These suites offered to 
program management a set of investment 
recommendations and roadmaps that 
would enable such mission to be 
accomplished. 


2 nd Gen 


Technology Assessment Process isat/aee 


Analysis Results 
Loss of Crew 
Vehicle Reliability 
Annual Ops Cost 
DDT&E Cost 
Performance 



Figure 1 IS AT Technology Assessment Process 


Another exercise that NASA led was the Next Generation Launch Technology (NGLT) Systems Analyses, 
which had a goal similar to that of the ISAT. The primary difference between these two activities is the 
vehicle architectures assessed and the methodologies associated with the mission analyses and the 
technology assessments. Hypersonic air-breathing vehicles were a component of these studies along with 
reusable rocket launch vehicles. Also, a technology assessment tool, Value Stream Technology Profiler 
(VSTeP), was added to the tool suite. VSTeP provided technology availability data as well as capability 
with its data being populated by industry, government, and academia technologists. The step forward is 
that the profiler used computer databases with online access enabling rapid manipulation of data for the 
required analyses. Additionally, response surfaces enabled high volume systems analyses, which allowed 
for statistical analyses of technology and mission analysis suites. 



Figure 2 NGLT Systems Analysis and Technology Assessment Process 
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Both of these studies represented dramatic leaps forward in tool integration, systems analysis and integrated 
vehicle analyses. These activities represented the vision of future technology assessments and portfolio 
planning. The computing and networking technologies were largely the limiting factors of the times. The 
next phase of technology planning and assessments, new tools will be needed that enable broad information 
searching, data manipulation, and decrease cycle analysis time. 


V. Expanding Technology Assessment Methodologies 

Technology is neither good nor bad; nor is it neutral - Melvin Kranzberg - Six Laws of Technology 
The Advanced Concepts Office began expanding its methods for technology assessment about three years 
ago with a focus on incorporating overall strategic mission objectives with engineering design concept trade 
studies. ACO identified three goals with regard to expanding our current technology assessment 
methodologies. 

• Improve stakeholders’ ability to make decisions regarding technology 

• Enable "information-based decisions” 

• Lead the effort to align MSFC’s organizational posture toward its corporate business objectives 
To accomplish these goals, ACO began incorporating the methods described in Ref. 16, Analyzing the 

Functional Dynamics of Technological Innovation Systems (TIS). Since TIS focuses more on policy and 
strategy, it mapped well with goal three: aligning organizational structure to business objectives. ACO then 
incorporated NASA’s Strategic Space Technology Investment Plan § (SSTIP), Technology Roadmaps, the 
National Resource Council assessment of these technologies 17 , the Human Exploration and Operations 
(HEO), Advanced Exploration Systems (AES) projects* **, as well as MSFC Engineering Departments, 
HEOMD Strategic Knowledge Gaps’ 5 ' 1 ' and other associated information. The first assessment was to compare 
an engineering trade study for the Dual Use Upper Stage (DUUS) concept against NASA’s overall Strategic 
Plan. The DUUS is described in Ref. 18. 


Figure 3 Comparison of Engineering Trade Study with NASA strategic Goals, shows a dashboard 
depicting the relationship between the DUUS engineering trade study using MCDA optimization methods 
and NASA’s SSTIP using TIS methods. This figure demonstrates that optimizing for design does not 
necessarily align with strategic goals, nor should it. It does demonstrate the contribution the DUUS element 
has on the overall strategy however. The dashboard was dynamic so that a decision maker could assess how 
technological criteria aligned against strategic goals. By adjusting the weight assigned to each criterion, one 
could balance (optimize) both the DUUS and strategic goals. In this particular case, the decision maker had 
to give up schedule and cost constraints to balance both goals. 



ACO also performed an assessment between a set of AES projects and the HEOMD Strategic Knowledge 
Gaps (SKG). For each project, the set of technologies the project was focused on was determined; a mapping 
was made between technologies and each SKG. A dashboard was built, to determine the alignment between 


§ http://www.nasa.gov/pdf/674740main_07-17_12DRAFT_Strategic_Space_Tech_plan.pdf 

** http://www.nasa.gOv/directorates/heo/aes/index.html#.VawM7ipViko 
http: //science.nasa.gov/media/medialibrary/20 1 2/05/04/HEOMD_Strategic_Knowledge_Gaps_— 
_Mike_W argo .pdf 
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each project and each knowledge gap. Figure 4 Assessing AES Projects and Strategic Knowledge Gaps shows 
the dashboard comparing the alignment between AES projects and SKG. From this information a decision 
maker can easily identify which knowledge gaps are not being closed, which knowledge gaps have multiple 
projects and which projects close multiple gaps. The decision maker can use this information as an assessment 
tool for project funding and to solicit focused research on hard to close gaps. 


Strategic Knowledge Gaps Associated with Potential Human Missions to the Martian System 



NASA’s Office of Chief Technologist 
approved a study led by ACO to facilitate an 
assessment of potential partnerships with 
potential commercial endeavors using 
Inspiration Mars (IM) as an example 
mission. Using the SSTIP, Technology 
Roadmaps and NRC assessments a radar 
map was created to provide an overall view 
of potential technology benefits. A set of 
top-level technologies were identified that 
would benefit both NASA and a commercial 
endeavor. Subject matter experts (SME) 
were interviewed across NASA centers for 
each top-level technology area to address 
specific top-level technologies. By 
performing the analysis first, specific 
technological benefits could be refined from 
the experts reducing man-hours. The 
dashboard is shown in Error! Reference 
source not found.. The radar plot was used 
to categorize top technologies, such as 
Autonomous Systems, Human Health, and 
Material/Structures, where a decision 
maker could easily identify which 
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Figure 5 Commercial Partnership Technologies 
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technologies were the most challenging (long read poles) and which technologies would be most beneficial 
to the mission. Three tables were created; High-Benefit/High-Challenges NASA focused, High-Benefit/Low 
Challenge (Commercial Focused) and Moderate Benefit/Moderate Challenges (Partnership). 

For another study, ACO considered the relationships between mission categories, the elements that 
support those missions, the subsystems that enable the elements, the organizational resources required to 
build the subsystems, the technologies that support this technological system and the scenarios that fund the 
missions. A dashboard was built so that a decision maker could see the impact one particular mission or 
funding scenario had on similar missions, elements, subsystems, resources and technologies. The intent was 
to analyze the structure of the technology system using a Technological Innovation System (TIS) framework, 
and organizational network modeling. A dashboard was built and is depicted in Figure 6 Dashboard Mission, 
Elements, Subsystems, Technologies and Resources. The case shown is for Human Mars Mission. Missions 
are then ordered based on the similarities or required elements, e.g. Human Mars, Human Lunar, Human 
Mars Moons and Human Near Earth Asteroid are similar mission categories, as are Robotic Missions. 
Mission Categories require similar elements. Also depicted is an ordered list of subsystems, e.g. Tank design 
and Cryogenic Fluid Management (CFM). Also shown is perceived the organizational participation required 
to support these missions. The radar chart is used to assess the mission alignment between the technologies 
identified in the Technology Roadmaps and supporting subsystems, e.g. Multifunctional Lightweight Tanks 
and Cryogenic/ Liquid Storable Tanks. 



Over the past three years, ACO has done approximately 15 studies incorporating the TIS methodology to 
support strategic decision-making based on the relationships from an underlying technological network. 
There are three common elements in performing these studies. 

• Matrix gymnastics 

• Textual data mining and expert validation 

• Dynamic and Interactive Visualization 


The next section discusses these items in a more formal manner. 
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VI. Technology Assessment Tools 

In the past three years, as ACO refined its methods and processes for combining the various information 
required to do technological assessments, it became apparent that a set of tools will allow ACO to be more 
efficient. ACO is attempting to address some of the challenges identified in Ref. 15. ACO is doing this by 
addressing the following: 

• Formalizing the language used to communicate Technological Assessment 

• Providing easier ways to access and manipulate data 

• Providing the ability to present results in an easily a posteriori manner 

Most analysis is currently done in spreadsheets and this unlikely to change any time soon. Spreadsheets 
provide a spectrum of capabilities to the analyst, e.g. matrix manipulation, statistics, optimization, sorting, 
mapping, filtering, data lookup, chart generation, input forms and a programming language. However, 
efficiency is contingent on an individual’s ability with spreadsheet gymnastics, understanding the statistical 
tools, optimization techniques, and “speaking the programming language.” Take into further consideration 
that “research has suggested that errors are prevalent in spreadsheets 19 ,” and the challenges identified by 
Olechowski, Eppinger and Joglekar, the set of tools need to enable verification, validation and repeatability 
of assessments, as well as, the ability to model the complexity associated technological structures and 
trajectories. 

A. Formalizing the Language 

At a fundamental level, the mathematical methods employed by TAPP are based on Category Theory, 
Graph Theory and Formal Concept Analysis. Category Theory consists of a collection of objects that are 
connected by arrows (morphisms) to form directed and undirected graphs 20 . A graph consists of nodes 
(objects) and edges (relationships between nodes). Graph theory is the study of graphs, which are 
mathematical structures, used to algebraically model the pairwise relations between objects 21 . Formal 
Concept Analysis (FCA) is “based on the mathematization of concept and concept hierarchy. 22 ” FCA focuses 
on the ordered sets of objects, object attributes and relationships to establish hierarchies based on lattices, 
ontologies (01 og) and partially ordered sets (poset). 

The motivation for identifying these mathematical frameworks is that it 1) formalizes the methods used 
in Technology Assessment and 2) formal methods can be programmed, e.g. contextual computing. In the 
Expanding Technology Assessment Methodologies Section of this paper, an engineering concept analysis 
was presented on the Dual Use Upper Stage (DUUS). Like most studies, a set of Figures of Merit (FOM) 
were assigned on a set of alternative technologies, where each technology was assessed its merit based on 
cost, schedule, performance, safety and relevance. The intent is to find the best engineering solution, based 
on the FOMs. The FOMs were weighted and ordered using the TOPSIS ^method. 

The relationships between elements and subsystems can be stated formally as follows: 

Using a graph notation, where () — []->() denotes (node) — [relationship]^ (node) 

(: ELEMENT) — [: DEPEND_ON]-*(: SUBSYSTEMS) 

(: ELEMENT (name: ‘DUUS’)) — [: DEPEND_ON] -> (: SUBSYSTEMS (name: ‘ Structures’ ) [: DEPEND ON)) 

(TA: TECHNOLOGY (name: ‘Aluminum Lithium Structure’,) 


FOM: [(Cost: 4), (Schedule: 8), ( Performance : 7), (Safety: 5), (Relevance: 4)]}) 
This is often done in tabular (matrix) form with equivalent information 


DUUS 

FOMS 

Technology 

Cost 

Schedule 

Performance 

Safety 

Relevance 

Aluminum Lithium Structure 

4 

8 

7 

5 

4 


In the strategic domain, relationships are focused on strategic investments and technology roadmaps. 
Strategic FOMS have a different set of criteria and weighting and decision-making is based on multiple 
objectives. §§ The relationship between technologies and investment policy can be stated formally as follows: 


** TOPSIS: Technique for Order of Preference by Similarity to Ideal Solution - multi-criteria decision 
analysis, assumes criterion is monotonically increasing or decreasing 

§§ Multi-objective optimization involves more than one objective function so a single solution does not exist. 
It usually leads to a set of solutions. A dynamic visualization of the Pareto front provides knowledge a 
posteriori. 
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(: TECHNOLOGY)— [: DEPEND_ON] ^INVESTMENT {Policy: SSTIP))—[: DEPEND_ON]^(: BENEFIT-RISK) 

(: ELEMENT (name: ‘DUUS’))—[: DEPEND_ON]^(: SUBSYSTEMS) ^[: DEPEND_ON] -> 

(12.2.1: TECHNOLOGY {name: ‘Lightweight Concepts’,) 

FOM: [{Benefit: 9), 

{Technical Risk: 9), 

{Sequencing: 1), 

{Effort: -5)]}) 

This too can be represented in matrix form. 

The benefit of describing the study in graph notation, is that the connection between both studies would be 
represented formally as follows: 

(: INVESTMENT {Policy: SSTIP))— [: INFLUENCES]^: TECHNOLOGY) <- [ ]— (: ELEMENT {name ‘ DUUS ’)) 
Then, one can easily create the relationships, without a lot of spreadsheet gymnastics, one can easily 
incorporate, merge and integrate studies, using the programming languages, and database available today. 
This is precisely what ACO is attempting to do with TAPP. Drag and drop concept analysis. 

Matrix Representation -> Graph Representation -> Dynamic Dashboard 

As shown in Figure 3 Comparison of Engineering Trade Study with NASA strategic Goals, the analyst and 
decision-maker were presented a dynamic visualization of the two Pareto Domains, Overall Strategic 
Objectives versus the DUUS Concept. The dynamic dashboard allowed an analyst or decision-maker to 
adjust the weights of the FOMS, dynamically changing the order of the technology alternatives or strategic 
objectives, visualizing the impact of changing policy or the impact of changing design. In the study there 
is a definite benefit with lightweight structures, but overcoming DDT&E timing between alternate 
materials was one of the biggest constraints. If the analyst had the ability to compare or analyze alternate 
lightweight materials across all mission systems, a system of systems approach, the benefits associated with 
lightweight materials could be better ascertained. The optimal solution for the DUUS element may not be 
the most efficient solution when compared to all elements across all missions. In category theory this would 
be called a “universal property” used to determine the most efficient solution. Universal properties become 
important when evaluating the Pareto Efficiency between domains. It is important to understand the 
morphisms of initial and final properties between the objects. In category theory, one make an abstraction 
of the DUUS, identifying an initial state and final state by analyzing the range properties have on changing 
the concept. This method is not new to 
analysts. Analysts call it commonality. 

ACO is also using data mining techniques, 
including topic mapping and natural 
language understanding to extract 
information from a variety of information 
stores, such as the NASA Scientific and 
Technical Information Program, NASA 
Technology Roadmaps, Internal Project, 

Design Reference, Trade Studies and other 
available information. The intent of this 
capability is to drill down as deep as possible to extract information that would be relevant to 
Technological Assessment. One of the Figure 7 Hierarchical Concept Analysis using Data Mining 

planned exercises is to assess how well 

natural language understanding can determine technology readiness levels from unstructured text. Another 
planned exercise is to categorize wireless sensor development by mining approximately 400 documents 
using a topic mapper, then to combine that information across mission systems to shape a strategy for 
wireless sensor development. Figure 7 Hierarchical Concept Analysis using Data Mining shows an attempt 
to merge radiation protection semantic fingerprint to the NASA technology structure. 
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VII. Conclusion 

This paper discussed the methods ACO is currently developing to better perform technology assessments 
while taking into consideration Strategic Alignment, Technology Forecasting, and Long Term Planning. The 
Advanced Concepts Office (ACO) at NASA, Marshall Space Flight Center is expanding its current 
technology assessment methodologies by developing a new framework called TAPP based on ACO’s history 
using various processes, methods and tools to perform Technology Assessments. The goal in developing this 
framework is to improve stakeholder’s ability to make decisions regarding technology and enable 
“information-based decisions. 

TAPP addresses three types of challenges. 

• System Complexity - challenges related to the complexity of the system, whether it be at the 
component level, architectural level or system as a whole (across missions) 

• Planning and Review - challenges related to the decision process and technological progression 

• Assessment Validation - challenges related to quality of the information, subjectivity and 
imprecision 

TAPP focuses on three areas to improve efficiency. 

• Formalizing the language used to communicate Technological Assessment 

• Providing easier ways to access and manipulate data 

• Providing the ability to visualize results in an informative and exploratory manner 

ACO is using the TAPP framework to develop new tools to quickly integrate concept studies and 
information, and put the results into the hands of other analysts and decision-makers. 
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